1. Field of the Invention
The present invention relates to an image processing apparatus, an image processing method, and a program, and particularly to an image processing apparatus, an image processing method, and a program that attain high-speed process for generating a high-resolution image.
2. Description of Related Art
As a technique for generating a high-resolution image from a low-resolution image, there is super resolution.
Super-resolution is a technique in which a pixel value of each pixel in one frame of high-resolution image is found from a plurality of low-resolution images having overlap to thereby reconstruct an image at a resolution beyond a resolution of an imaging element such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor). For example, the super-resolution is used when generating a high-resolution satellite photograph, or the like.
FIGS. 1 and 2 are diagrams showing a principle of super-resolution.
Reference characters a, b, c, d, e, f shown in the upper parts of FIGS. 1 and 2 denote pixel values of a high-resolution image (SR (Super Resolution) image) to be found from low resolution images (LR (Low Resolution) images) obtained by imaging a subject, that is, pixel values of respective pixels when the subject is pixilated at the same resolution as that of the SR image.
For example, in the case where a width of one pixel of an imaging element is equivalent to a width of two pixels constituting the subject, and thus, the subject cannot be captured at the resolution at it is, as shown in FIG. 1, in a left pixel among three pixels of the imaging element, a pixel value of A obtained by mixing pixel values of a and b is captured, in a central pixel, a pixel value of B obtained by mixing pixel values of c and d is captured, and in a right pixel, a pixel value of C obtained by mixing pixel values of e and f is captured.
Reference characters A, B, C denote pixel values of pixels constituting LR images obtained by imaging.
When the subject in a position shifted by a width of 0.5 pixel of the pixel constituting the subject with respect to a position of the subject in FIG. 1 due to camera shake is captured as shown in FIG. 2 (captured while being shifted) together with the subject of FIG. 1, in the left pixel of the three pixels of the imaging element, a pixel value of D obtained by mixing pixel values of half of a, entire b, and half of c is captured, and in the central pixel, a pixel value of E obtained by mixing pixel values of half of c, entire d, and half of e is captured. Moreover, in the right pixel, a pixel value of F obtained by mixing half of e and entire f is captured. Reference characters D, E, F also denote pixel values of pixels constituting LR images obtained by imaging.
The following formula (1) is given from the imaging results of the above-described LR images. By finding a, b, c, d, e and f from formula (1), respectively, an image of a resolution higher than that of the imaging element can be obtained.
                                          (                                                            1                                                  1                                                  0                                                  0                                                  0                                                  0                                                                              0                                                  0                                                  1                                                  1                                                  0                                                  0                                                                              0                                                  0                                                  0                                                  0                                                  1                                                  1                                                                                                  1                    /                    2                                                                    1                                                                      1                    /                    2                                                                    0                                                  0                                                  0                                                                              0                                                  0                                                                      1                    /                    2                                                                    1                                                                      1                    /                    2                                                                    0                                                                              0                                                  0                                                  0                                                  0                                                                      1                    /                    2                                                                    1                                                      )                    ⁢                      (                                                            a                                                                              b                                                                              c                                                                              d                                                                              e                                                                              f                                                      )                          =                  (                                                    A                                                                    B                                                                    C                                                                    D                                                                    E                                                                    F                                              )                                    (        1        )            
FIG. 3 is a block diagram showing a configuration example of an existing image processing apparatus 1 that generates an SR image by super-resolution processing using Back Projection.
The image processing apparatus 1 of FIG. 3 is provided in a digital camera, for example, to perform processing of a still image obtained by imaging.
As shown in FIG. 3, the image processing apparatus 1 is composed of super-resolution processors 110 to 112, a summing circuit 12, an addition circuit 13, and an SR image buffer 14.
LR0, which is an LR image obtained by imaging, is inputted to the super-resolution processor 110, and LR1 is inputted to the super-resolution processor 111. LR2 is inputted to the super-resolution processor 112. LR0 to LR2 are images obtained by continuously imaging, and have overlap in imaging ranges thereof, respectively. When imaging is continuously performed, normally, the ranges of the subject caught on the images of the imaging results are slightly deviated from each other due to camera shaking or the like, and thus, do not completely coincidence with each other, thereby resulting in partial overlap.
The super-resolution processor 110, based on LR0 and an SR image stored in the SR image buffer 14, generates a differential image representing a difference between them, and outputs a feedback value to the summing circuit 12. The feedback value is a value representing the differential image at the same resolution as that of the SR image.
In the SR image buffer 14, an SR image generated by the super-resolution processing performed immediately before is stored. If the processing has just started, and thus, no frame of SR image has not been generated, for example, an image obtained by upsampling LR0 to an image at the same resolution as that of the SR image is stored in the SR image buffer 14.
Similarly, the super-resolution processor 111, based on LR1 and the SR image stored in the SR image buffer 14, generates a differential image representing a difference between them, and outputs a feedback value representing the generated differential image to the summing circuit 12.
The super-resolution processor 112, based on LR2 and the SR image stored in the SR image buffer 14, generates a differential image representing a difference between them, and outputs a feedback value representing the generated differential image to the summing circuit 12.
The summing circuit 12 averages the feedback values supplied from the super-resolution processors 110 to 112, and outputs an image at the same resolution as that of the SR image, which has been obtained by averaging, to the addition circuit 13.
The addition circuit 13 adds the SR image stored in the SR image buffer 14 and the SR image supplied from the summing circuit 12 and outputs an SR image obtained by the addition. The output of the addition circuit 13 is supplied outside of the image processing apparatus 1 as a result of the super-resolution processing, and at the same time, is supplied to the SR image buffer 14 to be stored.
FIG. 4 is a block diagram showing a configuration example of the super-resolution processor 11n (super-resolution processor 110, 111 or 112).
As shown in FIG. 4, the super-resolution processor 11n is made of a motion vector detecting circuit 21, a motion compensating circuit 22, a downsampling filter 23, an addition circuit 24, an upsampling filter 25, and a reverse-direction motion compensating circuit 26.
The SR image read from the SR image buffer 14 is inputted to the motion vector detecting circuit 21 and the motion compensating circuit 22, and LRn obtained by imaging is inputted to the motion vector detecting circuit 21 and the addition circuit 24.
The motion vector detecting circuit 21 detects a motion vector with the SR image used as a reference, based on the inputted SR image and LRn, and outputs the detected motion vector to the motion compensating circuit 22 and the reverse-direction motion compensating circuit 26.
The motion compensating circuit 22 applies the motion compensation to the SR image based on the motion vector supplied from the motion vector detecting circuit 21, and outputs an image obtained by applying the motion compensation to the downsampling filter 23. The position of an object caught on the image obtained by applying the motion compensation becomes closer to the position of the object caught on LRn.
The downsampling filter 23 generates an image at the same resolution as that of LRn by downsampling the image supplied from the motion compensating circuit 22, and outputs the generated image to the addition circuit 24. Finding the motion vector from the SR image and LRn, and converting the image obtained by the motion compensation using the found motion vector into the image at the same resolution as that of LR image corresponds to simulating the image obtained by imaging based on the SR image stored in the SR image buffer 14.
The addition circuit 24 generates a differential image representing a difference between LRn and the image simulated in such a manner, and outputs the generated differential image to the upsampling filter 25.
The upsampling filter 25 generates an image at the same resolution as that of the SR image by upsampling the differential image supplied from the addition circuit 24, and outputs the generated image to the reverse-direction motion compensating circuit 26.
The reverse-direction motion compensating circuit 26 applies motion compensation in the reverse direction to the image supplied from the upsampling filter 25, based on the motion vector supplied from the motion vector detecting circuit 21, and outputs a feedback value representing an image obtained by applying the motion compensation in the reverse direction to the summing circuit 12. The position of the object caught on the image obtained by the motion compensation in the reverse direction becomes closer to the position of the object caught on the SR image stored in the SR image buffer 14.
There has been developed to try and address the known issues with image processing (See, e.g., “Improving Resolution by Image Registration”, MICHAL IRANI AND SHMUEL PELEG, Department of Computer Science, The Hebrew University of Jerusalem, 91904 Jerusalem, Israel, Communicated by Rama Chellapa, Received Jun. 16, 1989; accepted May 25, 1990).